This presentation

Quarto: an open-source scientific and technical publishing system

  • What is literate programming
  • Learn about the different possibilities:
  • Render to the output format you want
  • Outsource rendering with Continuous Integration/Development (CI/CD)

Literate programming is the practice of mixing code and descriptive writing in order to execute and explain a data analysis simultaneously in the same document.

William Landau Developer of {targets}


For a bioinformatician, pure programming is barely happening, we need reports/websites where content and formatting are disconnected

Even better to separate content and rendering, outsourced to continuous integration/development.

Rmarkdown

Predecessor of Quarto.

Version 1 in 2016, the idea was to have text and code in one document, execute code and let the final markdown converted to different formats by pandoc.

Jupyter notebook were also exploring the same concept. Rendering order led sometimes to some frustration

Markdown

  • Lightweight markup language with a simple syntax

HTML

<!DOCTYPE html>
<html>
<body>

<h1>This is a heading</h1>

<p>This is some text in a <b>paragraph</b></p>

<h2>This is a second level heading</h2>

<ul>
<li><a href="http://exa.com"><code>site</code></a>
<li><img src="https://images.computerhistory.org/revonline/images/500004391-03-01.jpg?w=200">
</ul>
</body>
</html>

Markdown equivalent

# This is a heading

This is some text in a **paragraph**

## This is a second level heading

- [`site`](http://exa.com)
- ![](https://images.computerhistory.org/revonline/images/500004391-03-01.jpg?w=200)

Linus and Tux

Quarto

Extended Rmarkdown. Languages and Integrated Development Environment agnostic

Positron

Why use Quarto ?

  • Write detailed analyses that render to beautiful reports
  • Ensure reproducibility
  • Mix programming languages (passing along variables)
  • Excellent documentation
  • Export a single (.qmd) document to various formats (PDF, HTML…)
  • Many extensions
  • Text file that can (should) be tracked by a version control system (like git)

Quarto pipeline

Computation

  • In a fresh and clean session
  • Chunks are evaluated
  • Sequentially from top to bottom

Example for knitr

  • Extracts chunks
  • Interpret/Run them
  • Formats results as markdown
  • Re-inject in the document (MD)

Pandoc

  • Converts markdown to the desired document
    • PDF
    • HTML
    • Word
    • EPUB

Quarto document: Structure

YAML header

  • Document common variables:
    • title, date, author, …
  • Define format and its specific options
  • Use Tab for autocompletion

Free text in Markdown

  • Markdown syntax to write your descriptions, remarks
  • Literate programming

Code chunks

Code to be interpreted

  • knitr for
  • jupyter for
  • QuartoNotebookRunner.jl package for julia
  • native support for Observable
  • Outputs (tables/plots) are inserted below each

Output examples 1/2

HTML

Word

Output examples 2/2

Typst

PDF (LaTex)

Equations using \(\LaTeX\) syntax

How to add equations

  • Enclose in $ for in-line equations

  • $a^2+b^2=c^2$ renders as \(a^2+b^2=c^2\).

  • Double ($$) for separate equations.

$$G_{\mu v}=8 \pi G (T_{\mu v} + \rho _\Lambda \ g_{\mu v}) $$

renders as:

\[G_{\mu v}=8 \pi G (T_{\mu v} + \rho _\Lambda \ g_{\mu v})\]

Reports

  • Automatic Table of content
  • Author metadata (affiliation, , )
  • Here HTML output, but PDF is only one variable modification away

YAML config file

  • Mandatory name: _quarto.yml, at project root
  • Target of quarto render
  • To configure:
    • Books
    • Websites
    • Manuscripts
_quarto.yml
project:
  type: website
  render:
    - "*.qmd"
    - "!resources/"
    - "!data/"

execute:
  freeze: auto  # re-render only when source changes

website:
  title: "bulk RNA-seq"
  site-url: https://misb.pages.uni.lu/rna-seq
  image: img/visualization.png
  description: "Snakemake dependency graph"
  search:
    location: navbar
    type: overlay
  navbar:
    background: light
    logo: img/visualization.png
    left:
      - file: index.qmd
        text: ""
      - file: slides.qmd
        text: " Lectures"
      - file: practicals.qmd
        text: " Practicals"
      - file: setup.qmd
        text: " Setup tutorial"
      - file: exams.qmd
        text: " Exams"
    right:
      - file: about.qmd
        text: " About"
  page-footer:
    left: |
      All content licensed under a 
        except for _source:_ statements.
    center: "Copyright © [Université du Luxembourg](https://www.uni.lu/) 2024. All rights reserved."
    right:
      - icon: facebook
        href: https://www.facebook.com/uni.lu
      - icon: linkedin
        href: https://www.linkedin.com/school/166650
      - icon: instagram
        href: https://www.instagram.com/uni.lu
      - icon: youtube
        href: https://www.youtube.com/user/luxuni
    border: true

format:
  html:
    page-layout: article
    code-block-border-left: true
    code-block-bg: true
    code-copy: true
    code-link: false # requires pkg downlit
    theme:
      light: cosmo
      dark: [cosmo, theme-dark.scss]
    fontsize: 1.2em
    css: styles.css
    toc: true
    lang: en
    date-format: long
    highlight-style: atom-one

Website example

With previous _quarto.yml and some CSS tweaks. Grid listing

Websites

  • Extensive use for teaching
  • Building and deployment via Continuous Integration (CI)
    • Gitlab
    • GitHub
  • Automatic listing pages, search engine, navigation bar and smart caching
  • Demo using the BASV53
    • Table listings

Books

  • Bundle large chapters into a website
    • Chapters on the left
    • ToC on the right
  • Quarto docs are one
  • Exports to PDF / EPUB etc…

Dashboards

Demo simple Quarto doc in VScode

Minimal header

---
format: html
---

---
format: pdf
title: Bioinfo DLSM team 2024
author: Aurélien Ginolhac
date: today
execute: 
  echo: false
---

Quarto Manuscripts

Command-line

 quarto create project manuscript dlsm-bioinfo
? Title (dlsm-bioinfo)  Services of the bioinfo DLSM team
Creating project at DLSM/Seminars/2024-quarto-intro/dlsm-bioinfo:
  - Created _quarto.yml
  - Created .gitignore
  - Created index.qmd
  - Created references.bib

Outcome

 tree dlsm-bioinfo/
dlsm-bioinfo/
├── dlsm-bioinfo.Rproj
├── index.qmd
├── _quarto.yml
└── references.bib

1 directory, 4 files

PLoS template

Command to add an extensions among many available (+165 references)

quarto add quarto-journals/plos

Quarto Manuscripts, example

dlsm-bioinfo/
├── bioinfo.qmd
├── dag-atac-seq.png
├── dag-rna-seq.png
├── insert-citation.png
├── orbi-bib-export.png
├── _extensions
│   └── quarto-journals
│       └── plos/
├── gaigneaux.bib
├── ginolhac.bib
├── references.bib
├── plos2015.bst <- bib style
├── _manuscript/ <- outputs
└── _quarto.yml  <- main config

Demo HTML output

PLoS PDF example:

Quarto extensions

  • We saw already some examples

Compiled here

Quarto Drop: a console in your browser

  • Add in header:
---
revealjs-plugins:
  - drop
---
  • Drop a console of webR or pyodide with ` or button.
  • Pre-load packages
format:
  revealjs:
    drop:
      engine: webr
      webr:
        packages:
          - ggplot2
          - dplyr

Example:

  • {ggplot2} gets automatically downloaded

Outsource rendering

  • Quarto output files usually rendered on your local machine.
  • For HTML files/website, those are static files.
  • Rendering could be done on someone else computer aka the cloud.
  • CI/CD (Continuous Integration/Continuous Deployment) can help
  • But you must use git , as you do anyway right?
  • Two examples:
    • GitHub Actions
    • Gitlab CI

CI description with Docker/git/github

GitHub Actions

.github/workflows/publish.yml
on:
  workflow_dispatch:
  push:
    branches: main
name: Quarto Publish
permissions: 
    contents: write
    pages: write
jobs:
  build-deploy:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository
        uses: actions/checkout@v4 
      - name: Set up Quarto
        uses: quarto-dev/quarto-actions/setup@v2
        with:
          version: pre-release
      - uses: r-lib/actions/setup-r@v2
      - uses: r-lib/actions/setup-r-dependencies@v2

      - name: Publish to GitHub Pages (and render)
        uses: quarto-dev/quarto-actions/publish@v2
        with:
          path: quarto-intro.qmd
          target: gh-pages
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
  • 3 steps:

Gitlab CI

  • More verbose, explicit
.gitlab-ci.yml
image: alpine:latest

stages:
  - build_image
  - build_site
  - deploy

build docker image:
  stage: build_image
  image: docker:20.10.12-dind
  before_script:
    - docker info
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY
  script:
    - 'docker pull $CI_REGISTRY_IMAGE:latest || true'
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  only:
    changes:
      - Dockerfile

build_site:
  stage: build_site
  image: $CI_REGISTRY_IMAGE:latest
  cache:
    key: $CI_JOB_NAME
    paths:
      - _freeze/
      - _site/
  artifacts:
    name: "$CI_JOB_NAME"
    expire_in: 2 days
    paths:
      - _site/
  before_script:
    # https://mastodon.social/@eddelbuettel/111693310455494722
    - |
      Rscript -e "read.dcf('DESCRIPTION', 'Imports') |> 
        tools:::.split_dependencies() |> 
        names() |> 
        setdiff(tools:::.get_standard_package_names()$base) |> 
        install.packages()"
  script:
    - quarto render
  when: always

Deploy to BASV53:
  stage: deploy
  only:
    - main
  dependencies:
    - build_site
  before_script:
    - 'which ssh-agent || ( apk add --update openssh )'
    - eval $(ssh-agent -s)
    - 'which rsync || ( apk add --update rsync )'
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - 'echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
    - rsync -azh -e "ssh -p 8022" --delete --filter 'protect /preview' $CI_PROJECT_DIR/_site/ $SSH_HOST:/var/www/basv53/

  • Failed pipeline: no deployment as Building site failed.
  • Most recent pipeline, I modified Dockerfile, thus 3 steps.

Dockerfile

Dockerfile
FROM rocker/r2u:22.04

RUN  apt-get update && \
  apt-get install -y \
  curl netbase \
  zip git \
  libxml2-dev libcurl4-openssl-dev libmagick++-dev \
  && rm -rf /var/lib/apt/lists/*

RUN install.r tidyverse rmarkdown biobase knitr V8 gt gtextras  ggimage ggrepel ggupset BiocManager && \
  installBioc.r deseq2 apeglm

# from https://quarto.org/docs/get-started/
ARG QUARTO_VERSION="1.6.33"
RUN curl -LO https://github.com/quarto-dev/quarto-cli/releases/download/v${QUARTO_VERSION}/quarto-${QUARTO_VERSION}-linux-amd64.deb && \
    apt-get update -qq && apt-get -y install \
    ./quarto-${QUARTO_VERSION}-linux-amd64.deb && rm quarto-${QUARTO_VERSION}-linux-amd64.deb \
    && apt autoremove -y && apt clean -y && rm -rf /var/lib/apt/lists/*

r2u is awesome, offering packages as APT.

  • CRAN as Ubuntu binaries, no compilation
  • {tidyverse} install is < 10 sec

by Dirk Eddelbuettel

Pre/Post render hooks

Tyler Morgan-Wall, author of rayshader

  • Replacing a Wordpress website to Quarto
_quarto.yml
project:
  type: website
  pre-render: "echo Let the rendering begin!"
  post-render:
    - "cp -rv posts/images/. _site/posts/images/"
    - "cp -rv posts/videos/. _site/posts/videos/"
    - "Rscript transfer_site.R upload_site"

Embed videos:

{{< video https://www.youtube.com/watch?v=U8_Dc_ru8fg height="350" >}}

Quarto Live

---
format: live-html
engine: knitr
---

```{webr}
for (x in 1:5) {
  print(x ** 2)
}
```

Warning

This is a temporary fix for knitr:

{{< include ./_extensions/r-wasm/live/_knitr.qmd >}}

Positron in beta since June 2024

Before we stop

Summary

  • What is Quarto (qmd)
  • Basic syntax of Markdown
  • Separate content from rendering
  • render to different output formats

Acknowledgments 🙏 👏

  • Anthoula Gaigneaux

  • Eric Koncina

  • Mine Cetinkaya-Rundel

  • Xie Yihuie

  • Hadley Wickham

  • Christophe Dervieux

  • Artwork by Allison Horst

  • Mickaël Canouil

  • Jenny Bryan


Thank you for your attention!